Agile Software Development

Agile software development methodologies provide many benefits including reduced time to market, lower defect rates, and more accurate requirements tracking. This paper describes Strategic Data Systems’ experience with agile software development, and demonstrates the advantages an organization may gain through implementing a similar development process. Introduction Ideas about the process of developing software have been around almost as long as software itself. These ideas progressed from ad hoc processes to various forms of the waterfall model. Then in 1999, Kent Beck published a book entitled Extreme Programming Explained. The ideas presented in this book broke away from the waterfall model and presented a new paradigm for developing software Agile Software Development. This paper focuses on the agile development process developed at Strategic Data Systems and presents the results of using this process on three projects. The most basic software development method is the ad hoc method. This method has been around since the earliest days of software development and is the method generally encouraged in undergraduate schools across the country. It has two simple stages: Write some code; Fix the problems in the code. No thought is given to the overall design, testing, or maintenance of the system until after the code has been written. Management often encourages ad hoc software development in an attempt to get products finished quickly. However, as studies have shown, products developed using this approach are of low quality. Agile Software © Strategic Data Systems 2004 3 Development A typical programmer will introduce 100 defects per thousand lines of code written using the ad hoc development process. An undisciplined approach to software development has three primary problems. First, after a number of iterations through the code and fix cycle, the code becomes so poorly structured that changes become increasingly difficult and expensive. Second, the software does not match what the user needs, and finally, code is expensive to fix because of poor preparation for testing and modification. As a matter of practice, no reproducible tests are developed. Unfortunately, the most common methodology still used today is the ad hoc, code-and-fix method. Applying some form of discipline will increase the quality of the code produced. As will be shown later, an advantage of agile software development is that it is a “lightweight” approach. When people are used to using no processes at all, simpler processes are more likely to be followed, and therefore produce better results. The traditional waterfall software development methodology was introduced to curb the problems of the code-and-fix method. 2 It provides a highly disciplined approach to developing software through a series of phases. A requirements-gathering phase, design phase, testing phase, and maintenance phase were added to alleviate some of the problems found in the ad hoc approach. The waterfall approach has become the basis of most software development models adopted in the industry. Studies have shown that a good software development process can greatly increase the quality of the software produced. A process can reduce defects from the 100 for ad hoc to 30 per 1000 lines of code. Agile Software © Strategic Data Systems 2004 4 Development However, this method has met with its own set of difficulties. One significant problem has been its emphasis on documentation. The waterfall method assumes that the user knows what is required of the system at the beginning of the project. Experience has shown that end users often do not know what the system should do in detail until they have worked with it. This is particularly true of interactive end-user applications (i.e. Web applications). In an attempt to alleviate this problem, user expectations may be documented using elaborate UML diagrams, yet the requirements are often misunderstood by the development team. The problem with the waterfall method is its inflexible partitioning of the project into these distinct stages. Delivered systems are sometimes unusable, as they do not meet the customer’s real requirements. – Ian Sommerville Many of the problems with the waterfall method have only recently been addressed with the agile development model. The agile approach takes much of the formal documentation out of the waterfall method and replaces it with frequent face-to-face communication and short user stories. Agile processes allow for changing requirements throughout the development cycle and stress collaboration between software developers and customers. Agile Software Development SDS has implemented a custom agile software development process. The “Agile Manifesto” established a common framework to create such a process: Value individuals and interactions over process and tools; value working software over comprehensive documentation; value customer collaboration over contract negotiation; value responding to change over following a plan. Agile processes considered during the development of the SDS process include Extreme Programming, Lean Development, Crystal, and Scrum. A set of principles, outlined in the Agile Manifesto, were also incorporated into the SDS process: Satisfy the customer through continuous delivery of most important software. Embrace changing requirements, even late in development. Ensure Developers and Business people work together throughout the development cycle. Employ only motivated people. Give them the work environment they need and trust them to get the work done. Realize communication is most efficient in face-to-face conversation. Primarily measure progress through working software. Continually pay attention to technical excellence and detail. Understand that simplicity – the art of maximizing the amount of work not done – is essential. Regularly evaluate the team and tune it for maximum productivity. In developing an agile software development process, one must consider two important ramifications of managing an ever-changing environment: While the goals are achievable, project details are often difficult to obtain; and that much of the detail of the project is not documented or repeatable. Note that these ramifications may be difficult for a seasoned project manager to work through. Organizations that employ rigorous methodologies view people differently from those that use agile methods. Rigorous methodologies are designed to standardize people to the process, thus making progress easier to track. Agile processes are designed to capitalize on each individual’s strengths, placing more responsibility on the person rather than the process. To be agile, one must balance flexibility and structure. A minimalist approach to development reduces costs through streamlining, Agile Software © Strategic Data Systems 2004 5 Development but even more importantly, it must incorporate a balance between chaos and order in which creativity and innovation may thrive. One of the main challenges with traditional methods of software development is that after the requirements-gathering phase is complete, customer interaction with the development team is limited until close to time of production. In this “big-bang” approach, it is difficult for the customer to determine whether progress is being made throughout the project. Agile processes call for significant interaction between the customer and the development team. Extreme Programming in particular requires a customer to be collocated with the developers. The SDS process fulfills this expectation by documenting user stories and conducting weekly planning meetings. User stories describe something the customer wants the system to do. The description should be detailed enough for a developer to understand and be able to estimate the length of time to implement the story. The estimate should be between one and five weeks; if the developer thinks it will take longer than five weeks to implement a story, the story should be broken down into smaller stories that fit in the oneto five-week range. At the weekly planning meetings, open user stories and open bugs are reviewed, priorities for the week are determined, and goals are set. In this way, the customer is kept current with the team’s progress, understands the team’s current challenges, and has an accurate picture of the progress of the system. The SDS process calls for new functionality and fixed bugs to be integrated and released weekly. In this way, the customer can see progress early and determine when feature function is ready for deployment. These small release cycles keep the team focused on the priorities determined during the weekly planning session. Generally, design in the SDS workflow takes place just before coding begins. Designs are kept simple; that is the design that sufficiently fulfills the user story and runs all the test cases. When the design is translated to code, careful attention is paid not to duplicate code. A developer will write test cases and implement the code at the same time. The work is kept within the developer’s intellectual control because he or she is continuously making small design and implementation decisions and increasing functionality at a relatively consistent rate. New functionality is not considered properly implemented unless these new unit test cases and every other unit test case written for the code base run without error. A typical development process will call for one person to be seen as the technical lead. This person tracks requirements, attends review and status meetings, and is often one of the better programmers on the project. The demands of project organization result in the lead struggling with communicating to the team all that the team needs to know for the project to be a success. At SDS, two people lead the team, each with a different focus. One person focuses on interacting with the customer, understanding the product, and generally making sure the team does the right thing. The other person focuses on the architecture and the technical quality of the system, and ensures things were done correctly. By distributing leadership responsibility, the leadership pair can Agile Software © Strategic Data Systems 2004 6 Development fully understand all aspects of the system and clearly communicate the priorities and architecture to the rest of the team. The SDS process also demands collective ownership of the source code. This means that anyone can change any piece of code at any point in time. This would be impossible except for the large collection of unit tests that can be run to ensure that a change in one module does not affect stable code in other modules. Since SDS promotes collective ownership of the source code, each team member’s changes need to be integrated as often as possible. To support this, the SDS process calls for the source code to be rebuilt and redeployed on an integration server each night. When the deployment is complete, all of the unit tests are run against the system each night to ensure the quality of the system. If the product does not build correctly or the unit tests do not pass, an email notification is sent to the team. Only the customer ultimately knows when the system fulfills the expectations of the envisioned product. The customer then plays an important role in validating new functionality and fixed bugs, and in determining when the product will ultimately move into production. The SDS process expects customer feedback at the beginning of each week, and at the end of an iteration. The agile process that SDS has adopted has proven to be very successful. The three projects developed in the SDS Agile Development Center report two defects per 1000 lines of code – a significant increase in quality compared with the 30 defects per 1000 lines of code reported on traditional projects. This significant increase in productivity is probably best explained by Brooks’ statement in The Mythical Man-Month that the difference between the best and worst programmers may be a ratio of 10:1. Agile Software © Strategic Data Systems 2004 7 Development A comparative study of four processes demonstrates the productivity gains in using an agile software development process. For this study, two standard measures are compared: defects per 1000 lines of code and lines of production code per developer per month. Studies show the average number of lines of code produced per month range from 250 to 700, and the number of defects per 1000 lines of code (LOC) range from 15 to100. The three projects developed in the SDS Agile Development Center report two defects per 1000 lines of code, and 1275 lines of production code per developer per month. This comparison validates the findings of Forrester Research that the core tenets of agile development – weekly reprioritization of requirements, short iteration cycles, highly skilled teams, and strong business participation – all lead to a successful product. Delivering high-priority requirements in shorter time frames is an organizational benefit. The increased team efficiency reduces both development and maintenance costs, and frees the development organization to address other projects. Agile Processes and RUP Many development organizations use and are interested in continuing to use the Rational Unified Process (RUP). Based on the findings above, an organization may want to incorporate agile methods with already established RUP-based methods. This topic has been thoroughly addressed by Robert Martin in the formulation of dX. RUP states that successful software starts with a vision statement. This document is developed by the customers of the system and serves three essential purposes: gain agreement on the problems needed to be solved, define the system boundaries, and describe the most important features of the system. This vision is documented using the Unified Modeling Language (UML). The important components of the system are represented with Use Cases, which make it easy for end users to articulate what is needed, and for the development team to validate that requirements are complete and accurate. RUP also calls for prototyping the user interface, which provides a general approach for gathering feedback on the user interface. Consistency of the user interface is essential to usability, and therefore this step is critical in developing Web-based applications. Four best practices of software development embodied in RUP fit very nicely with agile development processes: 16 Agile Software © Strategic Data Systems 2004 8 Development Develop Iteratively Iterative development is based on an approach that relies on continuous discovery, design, and implementation. It enables the project team to identify project risks as they occur and make appropriate decisions to mitigate these risks. Iterative development is at the heart of agile methods. Iterations are planned, worked, and released within a short period of time – one week to one month. Manage Requirements Requirements management is focused on managing and organizing changing requirements in a dynamic environment. Good requirements management provides a framework in which the customer can make changes to the system specification after development of the system has started. By fully utilizing the iterative development process and planning each iteration with the customer, the development team can efficiently manage requirements. When a new requirement is introduced, sets of user stories describing the requirement are documented. When the next iteration is planned, the customer has the full responsibility and freedom to choose this new requirement to be implemented if it has the highest priority. Utilize Component Architectures Architecture describes the structure of the application in terms of components and the fundamental mechanisms and architecture patterns that will be employed. A good architecture considers the flow of data from the data source to the customer, and the system requirements. When a good architecture is employed, obsolete software components can be replaced without significant effort. Agile processes fit well with this practice by calling for constant refactoring. Refactoring is the process a developer follows whereby the system is reworked to fully integrate changes into the system and eliminate a patchwork of subsystems. The best way to accomplish this practice is to employ a solid architecture that will support the constant change. Verify Quality The concept of quality focuses on both the product and the process followed to build the product. By simply adhering to a process, a team can decrease a system’s defect ratio by 70%. Both aspects of quality are supported by agile methods. Writing unit tests while the code is written increases the quality of the product. These unit tests are run each time a new feature is introduced to the code base, thus ensuring that introducing the new code did not break the system. One of the principles of the Agile Manifesto calls for the development team to evaluate itself often and make sure that it is following processes that support the development of a quality system. When problems are identified, new processes should be introduced. When a part of the process is deemed inefficient it should be changed or eliminated. Thus the process itself is always improving. Agile Software © Strategic Data Systems 2004 9 Development In its simplest form, RUP consists of some fundamental workflows that are similar to the waterfall method: 1. Business Engineering. Understanding the needs of the business. 2. Requirements. Translating business need into the requirements of a software system. 3. Analysis and Design. Translating requirements into a software architecture. 4. Implementation. Writing software that fulfills the architecture and business requirements. 5. Test. Ensuring that all of the requirements exist and are correct. 6. Configuration and change management. Keeping track of the different versions of the system components. 7. Project Management. Managing schedules and resources. 8. Environment. Setting up and maintaining the development environment. 9. Deployment. Everything needed to roll out the project. These activities are not necessarily separated in time. The best practice of iterative development calls for them to be executed concurrently throughout the lifetime of the project. However, as the project matures, the emphasis on certain activities increases or decreases, but all activities are allowed to execute at any time over the course of the project. Robert Martin formalized an agile process based on RUP called dX. This process consists of four fundamental workflows: 15 1. Inception Major use cases are identified and documented by the customer representative. Prototypes may be created to identify any shortcomings in the use cases and explore potential system architectures. 2. Elaboration Use cases continue to be written by the customer representative. The customer prioritizes each use case; generally, high-risk use cases are given priority. The customer representative plans iterations by selecting use cases that will be developed during that iteration. An iteration usually does not last longer than a week. The features to be implemented are analyzed and designed to fit into the system architecture. The developers may use UML diagrams or other modeling tools to facilitate a thorough understanding of the system. The design is then implemented in code. Two people must examine every line of code that is produced. This may be accomplished by having two developers work together, or by having someone review all code as it is integrated with the system. To ensure the quality of the system, a unit test is written before the code it must test. Developers alternate between writing test cases and production code. The tests and the code grow together. True to the agile development spirit, designs and code begin as simple as possible; complexity is added only when mandated by a use case. Developers may change any code that does not integrate well with the overall system architecture. The team owns all of the code, and therefore, any developer can change any piece of code at any time. By mandating that the unit tests are current, simply running the tests ensures that any problem introduced because of a change is caught. Integration takes place at least daily. Agile Software © Strategic Data Systems 2004 10 Development 3. Construction Because the architecture and the project plan grow together, the construction and elaborations phases are essentially the same. The difference between them is simply in the stability of the architecture and the project plan. As the project moves into the Construction phase, the team works with the customer representative to develop a release schedule. 4. Transition The Transition phase starts after the first release, which usually takes place very early in the project. At that time the system is live, though it will not contain most of the functionality. The team will continue to plan out iterations and releases. The iterations accumulate until a new release can be made live. The team works to make the release cycle as short as possible so feedback can be incorporated into the system more quickly. Traditional development methodologies can help deliver projects on time and budget, and give management a good handle on project status throughout. However, if an organization is moving to a new technology platform or a project’s requirements are uncertain, these will not deliver similar results. dX or another agile process may be what is required to make the project a success. Choices There are a few factors for an organization to consider when deciding whether or not it should implement an agile process: understanding of requirements, access to resources, estimated project duration, impact of risks, and most importantly, time to market. Agile approaches are good when your requirements are uncertain. In this situation, agile processes may be uncomfortable, but will be more effective. It is important for the customer to understand that following a predictive process when requirements change is risky. By embracing the change and planning for it, agile processes lower this risk. Note that in practice, most project plans that imply a guaranteed delivery date, cost, and functionality are either wrong, heavily padded, or both. Agile processes open up the current status of the project to a point that all stakeholders can see the current state of the project at any point in time through running the latest version of the code. Market pressure is an important factor when assessing the business value of using agile methods. Time to market can be the decisive factor for the success of a project. Under strong market pressure, a delay of the project completion leads to a loss of market share, which drastically decreases the business value of the project. As market pressure increases, the productivity advantages required strongly encourage choosing an agile process. A dynamic and ever-changing set of requirements may cause upper management to fear that a project will never be completed. When told that the project will continue as long as the customer identifies high-priority, high-value work, management becomes uncomfortable. Most decision makers like a model in which project budgets are approved and the project remains within the budget confines. However, with agile methods, management can release to production whenever they are ready because with each iteration, the code base has a higher value and is generally more stable than the previous version. Thus, since management sets the priorities, they have complete control of when the project is finished. Agile Software © Strategic Data Systems 2004 11 Development Once an organization has determined to use an agile process, the success of the team depends on three major factors: the skills of the people involved; the processes used and commitment level to them; and the management and reporting systems being in place. Other items to consider are the number of teams involved, whether an agile process will be used only for one special project, and the method of rolling out the process across an organization. Barry Boehm’s premise of using better and fewer people is central to an agile process. Agile processes strip nonessential activities from projects, leaving developers more time to develop highpriority features. Although the difference in productivity between the best and worst programmers on a team may exceed the documented ratio of 10:1, the productivity difference matters most when the programmers are working on tasks essential to software delivery. Productivity differences are irrelevant when the programmers are engaged in nonessential activities. One problem the development team may encounter is when a team member likes to create the process artifacts more than code. The other people on the team will quickly assess the value of these activities and will not adopt them if they do not support the overall development effort. Many organizations opt to pick and choose best practices from among the various agile methods. Each development team should review its development process and identify the unique requirements that will enable that team to succeed. The twelve principles of the Agile Manifesto can be useful guidelines for this review. Note also that the process should support the product under development. The process for developing a flight control system will be very different from that used in developing an inventory-reporting tool. Agile Software © Strategic Data Systems 2004 12 Development The management and reporting systems will vary greatly from one organization to the next. Since agile processes do not call for much documentation, a bureaucratic organization may require control process definitions, responsibility definitions, meeting structures and agendas, reporting formats, issue escalation procedures, information-sharing tools, and so on. These will all have to be customized for each organization, but keep in mind that the focus of agile software development is to get working code into the customer’s hands as soon as possible. This philosophy may not work well in an organization weighted down by too much formality. It is highly recommended that an organization that chooses to implement an agile development process start small and put in place only what is needed when it is needed. Too often an overzealous development team has spent too much time and money on setting up an environment, developing a process, or completing other unproductive tasks. The focus should always be on delivering highquality, high-value software as frequently as possible. If a team discovers after an iteration or two that a new development environment would facilitate this goal more easily, then they should work to set up that environment. However, the process and environment are secondary and meant to support the development of working code. In considering a formal software development process, an organization has many models from which to choose. When a development process is used, the quality of the software produced is significantly better than software developed without the use of a process. Agile software development provides a customizable framework for a disciplined approach to software development. Agile processes are lightweight, and therefore may be more readily accepted by developers unaccustomed to using any process. Agile processes have many other advantages in quality, productivity, and time to market, and must be considered as a remedy for any organization frustrated with their current results. Agile Software © Strategic Data Systems 2004 13 Development